From Lexical Semantics to Text Analysis

1 Motivation One of the major challenges today is coping with an overabundance of potentially important information. With newspapers such as the Wall Street Journal available electronically as a large text data base, the analysis of natural language texts for the purpose of information retrieval has found renewed interest. Knowledge extraction and knowledge detection in large text databases are challenging problems, most recently under investigation in the TIPSTER projects funded by DARPA, the U.S. Department of Defense research funding agency. Traditionally, the parameters in the task of information retrieval are the style of analysis (statistical or linguistic), the domain of interest (TIPSTER, for instance, focuses on news concerning micro-chip design and joint ventures), the task ((lling database entries, question answering, etc.), and the representation formalism (templates, Horn clauses, KL-ONE, etc.). It is the premise of this paper that much more detailed information can be gleaned from a careful linguistic analysis than from a statistical analysis. Moreover, a successful linguistic analysis provides more reliable data, as we hope to illustrate with this paper. The problem is, however, that linguistic analysis is very costly and that systems that perform complete, reliable analysis of newspaper articles do not currently exist. The challenge then is to nd ways to do linguistic analysis when it is possible and to the extent that it is feasible. We claim that a promising approach is to perform a careful linguistic preprocessing of the texts, representing linguistically encoded information in a task independent, faithful, and reusable representation scheme. We propose a representation scheme, MTR 1 (for Minimal Text Representation), that does not constitute a text interpretation (nor does it \extract" or \detect" any particular information) but rather forms a common intermediate representation that must be further processed with the particular domain, task, and representation formalism in mind. The beneet of an intermediate representation at the level of MTR is that certain computationally expensive linguistic analyses do not have to be reduplicated for diierent tasks. The introduction of an intermediate representation (that has to be further evaluated) that also supports partial representation and thus incremental analysis of texts enables the 1 MTR is described in more detail in Bergler, 1992]. Here, we will only describe one aspect, proole structure.